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Evaluating E.S.E.A. Projects 
for the Disadvantaged 



J. RICHARD HARSH 




AS THE programs under the 
various titles of the Elementary and 
Secondary Education Act begin a scconu 
year, there is hardly an educator in the 
United States who would not exclaim, 
“This has been the busiest period of my 
professional life!" Most would agree this 
escalated pressure of activities has re- 
sulted from the pressing deadlines to 
write projects, obtain staff and mate- 
rials, and organize new programs, while 
at the same time maintaining the regu- 
lar educational program. 

At present it is important that educa- 
tors take time to make a “scorecard” 
of the process and products of the special 
project efforts to date. This appraisal 
should seek to review what has actually 
occurred in the first inning of the special 
projects. A review of the targets for ac- 
complishment and the means to get there 
in relation to the results which have been 
attained should provide some illumina- 
tion for developing better programs in 
the succeeding innings of the special 
projects. Hopefully, this would aid us in 
avoiding the characteristics of the light- 
ning bug, “. . . who is brilliant, but 
hasn’t any mind, for he blunders through 
existence with his headlights on behind.” 

February 1967 



Scoring Criteria: 

Steps in Evaluation 

One approach to developing a score- 
card or appraisal of the processes and 
products of the E.S.E.A. programs to 
date is to utilize the five steps of the 
evaluation process found in any explicit 
and systematic study. Each project may 
be considered in relation to how well its 
development and progress have met 
these five essential steps. Such an ap- 
praisal should provide important sug- 
gestions regarding the conceptualiza- 
tion of the project and the ways in which 
it may be improved. 

Step One — What was stated as the 
purpose of the project? This step in 
the evaluation process is often com- 
pleted with a minimum of effort and 
concern. Part of this is due to the fact 
that broad goals or objectives may be 
stated in sufficiently general terms so 
that they form an umbrella to encom- 
pass great diversity in process and prod- 
uct. Moreover, the statement of goals 
and objectives rarely differentiates 
among means, short-term educational 
experiences, ultimate achievements or 
outcomes. 
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Obviously it is desirable to refine and 
make objectives as specific and explicit 
as possible for the purposes of the study. 
However, statements of goals and objec- 
' : "cs arc often written to encompass a 
wide range of activities and pupil char- 
acteristics. Broad goals or objectives 
frequently become latch-strings to am- 
biguity and diffusion concerning the na- 
ture of the phenomenon that is to be 
studied. This condition creates the ne- 
cessity for the second step in the evalua- 
tion process. 

Step Two — IIow can I recognize the 
outcomes if I see them?: Definition of 
objectives in behavioral terms. This step 
in the evaluation process is a rigorous 
and demanding activity, for it requires 
that objectives or anticipated outcomes 
be defined and described in terms of be- 
haviors that may be observed, recorded 
and analyzed. For example, the objec- 
tive, “to develop and enrich the lan- 
guage of the disadvantaged,” offers am- 
biguous direction for data gathering or 
assessments. The test of behavioral defi- 
nition is to find consensus that the be- 
havior’s presence or absence represents 
attainment or non-attainment of the 
objective. 

Step Three — What is going to be done 
to achieve the objective?: The delinea- 
tion of situations and experiences 
through which the attainment of the ob- 
jectives of the program is to be accom- 
plished. This step in the evaluation proc- 
ess is frequently given more general than 
specific treatment. The situations in 
which the objectives are to be attained 
are often not examined in relation to 
their relevance for producing the de- 
sired behavior. Moreover, it is not un- 
common to find that description of the 



process, situation or environment of an 
educational program fails to identify 
specific methods, personnel or materials. 
This step in the evaluation process also 
demands rigor in order to give specificity 
to the “what,” “when,” “why” and 
“how” of the educational experience. 

Step Four — What information will be 
gathered as evidence of the attainment 
of the objectives?: The development 
or selection of appropriate assessments 
of the behavioral objectives. The effec- 
tiveness of evaluation depends on the 
relevance and reliability of the data 
accumulated concerning the program. 
The selection of data-gathcring devices 
should be directed by careful anal- 
ysis of the anticipated behaviors that arc 
defined in Step Two of the evaluation 
process. Too often general and “vener- 
able” measures of achievement which 
have titles as tests of reading, language, 
math or interest, etc., are assumed auto- 
matically to provide a measure of any 
and all behaviors that relate to that do- 
main of development. 

It is essential that data gathering for 
programs dealing with diverse student 
populations and innovative programs 
should not be restricted to measures 
previously standardized on normal pop- 
ulations within traditional programs. 
The items found in standardized tests 
are usually selected because of their 
“general fit” to typical student popula- 
tions in general areas such as reading 
or language. The studies by Frazier, 
Loban and others have indicated that 
language and reading have many dimen- 
sions with diverse relevance to various 
populations. Obviously, the generalized 
definitions of the materials which make 
up a content-domain may vary in dif- 
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lorcnt localities and may also be inap- 
propriate for diverse populations. As- 
sessments must be pertinent to the state- 
ment of objectives and the definitions of 
behavior to be attained. 

Unfortunately, assessment has fre- 
quently been perceived to mean only 
standardized testing. In contrast, as- 
sessment includes many forms of data- 
gathcring procedures. Observation is a 
form of assessment which includes the 
same properties of concern as any 
standardized test— reliability, standard 
error, validity, etc. Probably observa- 
tion has greater flexibility for gathering 
data concerning specific behavioral 
objectives than general standardized 
assessments designed to give generalized 
appraisals of content-domain of the cur- 
riculum. There are some who would 
decry that observation lacks sufficient 



objectivity and reliability to be consid- 
ered as an adequate form of data gath- 
ering; however, it should be argued 
that a systematic development of an 
observational process , which explicitly 
prc-defincs the time, circumstance, situ- 
ation and behavior that will be observed 
may have as high reliability and as 
great validity for the questions that arc 
to be judged or evaluated as some of 
the standardized instruments that lack 
the precision of “fit" to the particular 
objectives of the program. 

Another very direct form of assess- 
ment is the analysis of students’ prod- 
ucts. Such an analysis done in a sys- 
tematic manner, with the firsthand data 
of the productions from children, may 
offer evidence of the attainment of be- 
havioral objectives. Some are loath to 
consider this because involved in prod- 
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INGUiRY, EXPERIMENTATION, INVESTI- 
GATION Here's "Process , Science” in a 
new elementary Science Series, Grades 1-6, 
that provides unlimited fresh opportunities 
and sets up an environment for pupils to 
become personally involved in science 
through experiments and open-end 
questions./ 

This process-centered, inquiry series con- 
sistently invites the pupil "to think like 
a scientist." It develops and uses the tech- 
niques of science based on analysis, ob- 
ervation, experimentation and careful 
thought. 

We incite you to evaluate this fascinating 
new approach to science instruction. See 
the new LA1QLAW SCIENCE Series. 
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uct analysis is always the need to de- 
velop criteria by which the products 
may be judged. However, with explicit 
behavioral objectives, criteria for judg- 
ing the levels of adequacy or attainment 
in relation to the stated or anticipated 
behavioral outcomes may be made. 

Step Five — The fifth step in the 
evaluation process deals with the sum- 
marizing and analysis of data (and the 
judgments or evaluation of the out- 
come). It has been observed that there 
is a natural tendency for many to ap- 
proach the evaluation process by de- 
fining broad objectives and then jump- 
ing to the fifth step of making summary 
conclusions concerning the outcomes or 
products of the activity. 

In the fifth step of the evaluation 
process, there is need for rigorous exami- 
nation of the adequacy of the data 
which have been accumulated and sum- 
marized to represent attainment of the 
behavioral objectives described in Step 
Two of the evaluation process. The 
basic question is concerned with whether 
the data accumulated provide accept- 
able evidence upon which judgments or 
evaluations can be made. Furthermore, 
the data must also be examined in re- 
lation to adequacy for reflecting be- 
havioral changes as a result of the situ- 
ations designed for such attainment; 
perhaps the change in behavior may 
have resulted from concomitant factors 
unrelated to the particular project and 
design. 

After a Scorecard, 

What Action May Be Taken? 

Each of us concerned with education 
should consider the projects we are 
engaged in from the standpoint of how 



they may substantially contribute to 
further understanding of education and 
more effective learning of the students. 
Many members of the Association for 
Supervision and Curriculum Develop- 
ment are at this time directly or in- 
directly involved in some federal, state, 
local or professionally-sponsored proj- 
ect with research potential. 

What might each of us do to increase 
the probabilities that the new projects 
and research studies will make a sub- 
stantial contribution to the development 
of education? As we reflect upon our 
own attitudes and reactions to research 
and evaluation studies we have re- 
viewed, the following guidelines or 
commitments for future activities are 
suggested: 

1. To budget some time to review 
critically the adequacy with which the 
five steps of the evaluation process have 
been used. 

2. To persist in defining the (a) con- 
tent and process of the activity, (b) 
characteristics and circumstances of the 
learners, (c) location and situation for 
learning, (d) behavioral objectives or 
results anticipated until these are com- 
municable to participants and observers 
with common meaning. For example, 
use the simple two-way test: If you and 
I observe the aspects or resultant be- 
haviors of a project, will we be able to 
perceive that which is said to be occur- 
ring in the report? 

3. To clearly differentiate planning 
and developmental experiences from 
investigations designed to determine the 
outcomes of a prescribed educational 
experience. 

4. To insist that evidence to support 
or refute a research hypothesis or a 
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behavioral objective is composed of 
data with demonstrable reliability and 
validity. 

5. To make. « vplicit descriptions of 
the sampling procedure by which in- 
dividuals and products were assessed 
in the data-gathcring process. 

6. To provide for the presentation of 

data in the form and units by which 
the data were gathered; e.g., incidents 
of observed behavior, quantity and 
quality of product or test performance 
should be presented in the units of 
measurement rather than summarized, 
converted, equated or interpreted by 
“perceptions of previous behav: or 

“norms,” or comparisons with other 
general but unassessed groups. Such 
pooling of data, conversions, or inter- 
pretations provoke questions or uncer- 
tainty as to the basis upon which the 
situation is to be judged. 

7. To recognize and identify the 
multiple variables that are present in 
human behavior and devote special 
attention to the possibilities of plan- 
ning cooperative or related studies 
which may control or examine the ef- 
fects of various combinations of de- 
pendent variables. 

8. To give special attention to deter- 
mining the relevance of “venerable as- 
sessments” constructed and nonned in 
another era for data-gathering in new 
programs of today. Common usage and 
tradition tend to promote “face validi- 
ties” which may be erroneous, partial 
or distorted snapshots of what presently 
exists. 

There are numerous examples in the 
current periodicals of education being 
analyzed and evaluated by nonprofes- 



sional educators. Frequently the “case 
of one child” or the lacks of an adult 
population are cited as proof of the in- 
adequacy of education. Insofar as the 
goals and objectives remain in broad, 
ambiguous and unbehaviorally defined 
terms, each person from any frame of 
reference may make evaluations on the 
basis of his unique and undefined point 
of view. 

In contrast, the opportunities af- 
forded by the special projects should 
result in more extensive information 
concerning the description, prediction 
and control of the course of education. 

In the year 2000 when the inquiring 
student consults the reviews of educa- 
tional research of the 1950s and 1960s, 
he will undoubtedly be impressed with 
the tremendous quantity of research 
projects during the 1960s. When we 
pause, as we must, to consider the status 
of these various new projects, we should 
give some thought to the question, “Can 
we anticipate that the reviewer of edu- 
cational research done in the 1960s will 
find some substantial contribution to 
the understanding of human behavior 
and the learning process?” 
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